Surprising Results from a Measurement Study - Is Software Measurement an Exaggerated and Over-emphasised Area?

Many software companies are involved in eager attempts to manage experience and knowledge on an organisational level. Experience that can be used for estimating and planning purposes are of great value. One obvious solution can be defining and storing a set of measurements representing these experiences, carry out statistical analyses, and developing rules and guidelines to find and use this explicit knowledge. This four-year study using student projects indicates that the great majority of these measurements, seemingly unambiguous, include several uncertainties. I found it difficult to generalise the measurements since many factors have a different influence on each. The experience database I designed contains only three planning constants. The part of the study adding significant value to process improvement efforts was the analysis of software faults found in testing. Software faults were analysed for description, consequence, and proposals for actions to avoid similar faults reoccurring in the future. The potential for improvement resulting from the actions proposed is deemed high, and can probably reduce the number of major faults later on. Statistical analyses did not, however, prove useful because of all the uncertainty factors. Detailed analysis, primarily performed in seminars and face-to-face discussions, was necessary to interpret and understand the measures and statistical analyses. I am sceptical about using statistics due to all the difficulties associated with interpreting each measure, and since it is too tricky to perform a complete and fully satisfactory analysis of all existing uncertainty factors.